(feat) Add support for no KMS with s3-compatible backend by MonkeyCanCode · Pull Request #3501 · apache/polaris

MonkeyCanCode · 2026-01-22T04:54:59Z

This PR addressed one of the issues reported in #3440 where when end-user is non-AWS S3-compatible backend and have region set on the catalog property. With current code, we are determining if a S3 backend if AWS or not based on checking if account id and region are both set which is a bit problematic IMO. The account id here is derived from IAM role and the region is be set implicitly. That being said, when an user is trying to use assume role to auth and interact with S3-compatible storage, it can cause good amount of confusion as our current code base will add wildcard KMS policy to it if the backend is "AWS" (in this case, if a region and account id are both present...and region itself is valid for MinIO AFAIK and it default to us-east-1).

As discussed in #3496, instead of fixing the check for isAwsS3, we will proceed with a new catalog config instead.

Sample outputs:

# setup
./polaris --profile dev catalogs create --storage-type s3 --role-arn "arn:aws:iam::123456789012:role/polaris-warehouse-access" --default-base-location "s3://analytics-bucket/warehouse/" quickstart_catalog
./polaris --profile dev catalogs create --storage-type s3 --role-arn "arn:aws:iam::123456789012:role/polaris-warehouse-access" --default-base-location "s3://analytics-bucket-no-kms/warehouse/" --no-kms quickstart_catalog_no_kms

# outputs
./polaris --profile dev catalogs get quickstart_catalog | jq
{
  "type": "INTERNAL",
  "name": "quickstart_catalog",
  "properties": {
    "default-base-location": "s3://analytics-bucket/warehouse/"
  },
  "createTimestamp": 1769052561792,
  "lastUpdateTimestamp": 1769052561792,
  "entityVersion": 1,
  "storageConfigInfo": {
    "storageType": "S3",
    "allowedLocations": [
      "s3://analytics-bucket/warehouse/"
    ],
    "roleArn": "arn:aws:iam::123456789012:role/polaris-warehouse-access",
    "allowedKmsKeys": [],
    "stsUnavailable": false,
    "pathStyleAccess": false,
    "kmsUnavailable": false
  }
}

./polaris --profile dev catalogs get quickstart_catalog_no_kms | jq
{
  "type": "INTERNAL",
  "name": "quickstart_catalog_no_kms",
  "properties": {
    "default-base-location": "s3://analytics-bucket-no-kms/warehouse/"
  },
  "createTimestamp": 1769052562086,
  "lastUpdateTimestamp": 1769052562086,
  "entityVersion": 1,
  "storageConfigInfo": {
    "storageType": "S3",
    "allowedLocations": [
      "s3://analytics-bucket-no-kms/warehouse/"
    ],
    "roleArn": "arn:aws:iam::123456789012:role/polaris-warehouse-access",
    "allowedKmsKeys": [],
    "stsUnavailable": false,
    "pathStyleAccess": false,
    "kmsUnavailable": true
  }
}

Mailing list: https://lists.apache.org/thread/pksdwpzl0243ny1rvr8w4p6yz1m0xssn

Checklist

🛡️ Don't disclose security issues! (contact security@apache.org)
🔗 Clearly explained why the changes are needed, or linked related issues: Fixes #
🧪 Added/updated tests with good coverage, or manually tested (and explained how)
💡 Added comments for complex logic
🧾 Updated CHANGELOG.md (if needed)
📚 Updated documentation in site/content/in-dev/unreleased (if needed)

dimas-b

LGTM 👍 Thanks, @MonkeyCanCode !

Since this PR contains a REST API change, I believe we ought to open a dedicated dev ML discussion thread for it too (if only as a review notification) and have (lazy) consensus there before merging.

I'm approving in GH proactively.

MonkeyCanCode · 2026-01-22T21:37:29Z

LGTM 👍 Thanks, @MonkeyCanCode !

Since this PR contains a REST API change, I believe we ought to open a dedicated dev ML discussion thread for it too (if only as a review notification) and have (lazy) consensus there before merging.

I'm approving in GH proactively.

Sounds good. ML had being added to the description.

MonkeyCanCode · 2026-01-23T19:44:34Z

Thanks for review @dimas-b . I will have this up until next Monday as we collecting more feedback from mailing list as well as allow more review time as it is API change.

* Replace custom token-bucket implementation with Guava's `RateLimiter` (apache#3507) Addresses the issues discussed on the dev mailing-list discussion https://lists.apache.org/thread/gkyw7m4fcbjbzhcrlrp4kcq5lr05r0m4, opting to use Guava as the easiest replacement here. * Move idempotency_records schema to v4 and add H2 support (apache#3386) * Move idempotency_records schema to v4 and add H2 support * address comments and fix test failures * fix format * add comment to resource_id * (nit): Getting started examples with mc/s5cmd to aws cli (apache#3526) * Switch mc/s3cmd to aws cli * Switch mc/s3cmd to aws cli * Add support for no KMS with s3-compatible backend (apache#3501) * chore(deps): update amazon/aws-cli docker tag to v2.33.7 (apache#3558) * Update doc for helm around rateLimiter (apache#3562) * Disable renoavte update for python version (apache#3560) * Fix the Keycloak getting-started example for 26.5+ (apache#3568) The example was failing because Keycloak 26.5 introduced stricter validation rules for session lifespan and timeout. * NoSQL: Add to runtime-service (apache#3396) * NoSQL: Add to runtime-service This change adds the NoSQL persistence to polaris-runtime-service. * chore(deps): update amazon/aws-cli docker tag to v2.33.8 (apache#3575) * Add spark sql integration test for Hudi (apache#3194) * Fix ozone getting started example (apache#3574) * Fix Ozone getting started example * Fix Ozone getting started example * Change AWS CLI image to weekly (apache#3578) * fix(deps): update dependency com.diffplug.spotless:spotless-plugin-gradle to v8.2.1 (apache#3576) * chore(deps): update registry.access.redhat.com/ubi9/openjdk-21-runtime docker tag to v1.24-2.1769108682 (apache#3588) * removed references of BEFORE/AFTER_COMMIT_VIEW (apache#3554) * nits - post-merge fixes * Last merged commit 2b0ca21 --------- Co-authored-by: Huaxin Gao <huaxin.gao11@gmail.com> Co-authored-by: Yong Zheng <yongzheng0809@gmail.com> Co-authored-by: Mend Renovate <bot@renovateapp.com> Co-authored-by: Alexandre Dutra <adutra@apache.org> Co-authored-by: Rahil C <32500120+rahil-c@users.noreply.github.com> Co-authored-by: Innocent Djiofack <djiofack007@gmail.com>

Add support for no KMS with s3-compatible backend

424c333

github-project-automation bot added this to Basic Kanban Board Jan 22, 2026

github-project-automation bot moved this to PRs In Progress in Basic Kanban Board Jan 22, 2026

MonkeyCanCode mentioned this pull request Jan 22, 2026

DO-NOT-MERGE (fix) Refined AWS vs non-AWS detection for S3-compatible storage #3496

Closed

6 tasks

MonkeyCanCode requested review from HonahX, dimas-b and snazy January 22, 2026 04:55

dimas-b previously approved these changes Jan 22, 2026

View reviewed changes

github-project-automation bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jan 22, 2026

MonkeyCanCode changed the title ~~Add support for no KMS with s3-compatible backend~~ (feat) Add support for no KMS with s3-compatible backend Jan 22, 2026

Fix conflict on CHANGELOG.md

8a02928

MonkeyCanCode dismissed dimas-b’s stale review via 8a02928 January 23, 2026 18:39

MonkeyCanCode requested a review from dimas-b January 23, 2026 19:15

dimas-b approved these changes Jan 23, 2026

View reviewed changes

MonkeyCanCode merged commit 23b5070 into apache:main Jan 26, 2026
15 checks passed

github-project-automation bot moved this from Ready to merge to Done in Basic Kanban Board Jan 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(feat) Add support for no KMS with s3-compatible backend#3501

(feat) Add support for no KMS with s3-compatible backend#3501
MonkeyCanCode merged 2 commits intoapache:mainfrom
MonkeyCanCode:no_kms

MonkeyCanCode commented Jan 22, 2026 •

edited

Loading

Uh oh!

dimas-b left a comment

Uh oh!

MonkeyCanCode commented Jan 22, 2026 •

edited

Loading

Uh oh!

MonkeyCanCode commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MonkeyCanCode commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

dimas-b left a comment

Choose a reason for hiding this comment

Uh oh!

MonkeyCanCode commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MonkeyCanCode commented Jan 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MonkeyCanCode commented Jan 22, 2026 •

edited

Loading

MonkeyCanCode commented Jan 22, 2026 •

edited

Loading